Skip to main content

Common Tasks for Developers

This page gives checklists for things developers are typically asked to do in the infrastructure-automation repo. You make the change and open a pull request. An admin reviews, merges and deploys it (see Making Changes).

For the full explanation of each step, see Adding an App to Services User or Staff.

Always work on a branch​

Branch, pull request, admin as reviewer

Never commit or push directly to main in infrastructure-automation. Every change on this page goes through the same three steps:

  1. Create a branch from an up-to-date main: 
    cd infrastructure-automation
    git checkout main
    git pull
    git checkout -b <short-description>    # for example add-myapp-staff or myapp-new-env-var
  2. Commit your changes and push the branch: 
    git add <files you changed>
    git commit -m "<what you changed and why>"
    git push -u origin <short-description>
  3. Open a pull request on Gitea from your branch into main, and add an admin as a reviewer (the Reviewers box in the pull request's sidebar). Say in the description which app it's for and anything the admin needs to know, such as a Vaultwarden note you've created or edited.

Nothing reaches production until an admin has reviewed, merged and deployed the pull request. Merging alone doesn't deploy anything. Wait for the admin to confirm it's live before relying on the change.

Adding a new staff app​

Staff apps are internal tools, and back-end services that other apps call. They run on services-staff (10.0.0.19).

  1. Create ansible/hetzner-main/services-staff-apps/<app>/docker-compose.yml. The directory name is the app's name everywhere else.

  2. Add - <app> to staff_apps in ansible/hetzner-main/setup-services-staff.yml.

  3. If it needs secrets:

    • create a Vaultwarden Secure Note called exactly <app> .env in your project's collection;
    • add - <app> to staff_apps_with_env in the same playbook;
    • add env_file: .env to the service in the compose file.

  4. If it has a public hostname, add an nginx vhost in networking/nginx-config/<app>.abair.ie (see Public apps).

  5. Add the DNS record in Cloudflare (public apps only), or ask an admin to add it if you don't have Cloudflare access. Mention it in the pull request so it isn't forgotten:

    • Type: A
    • Name: <app> (giving <app>.abair.ie)
    • IPv4 address: 136.243.40.234 (the Proxmox host, which forwards web traffic to the webserver)
    • Proxy status: Proxied (orange cloud)

    The site isn't reachable until this record exists, even after the deploy. See Add the DNS record.

  6. Commit to a branch, push, and open a pull request with an admin added as reviewer (see above). Wait for the admin to confirm it's deployed.

  7. Check your app's CD workflow, and re-run it if needed. Once the admin confirms the deploy, open the Actions tab of your app's own repo (GitHub or Gitea, wherever its CI runs) and look at the latest run on main:

    • Passed: the latest image is deployed.
    • Failed: if it ran before the admin's deploy, its deploy step will have failed, because the app's directory (/opt/apps/<app>) didn't exist on the server yet. Re-run it (Re-run all jobs on GitHub, Re-run on Gitea). If it fails again after the admin's deploy, send the admin the error from the failed step.

    Then open the app in a browser (or call it, if it's internal-only) to check it's working.

Adding a new user app​

User apps are used by the public. They run on services-user (10.0.0.18).

  1. Create ansible/hetzner-main/services-user-apps/<app>/docker-compose.yml.

  2. Add - <app> to user_apps in ansible/hetzner-main/setup-services-user.yml.

  3. If it needs secrets:

    • create a Vaultwarden Secure Note called exactly <app> .env in your project's collection;
    • add - <app> to user_apps_with_env in the same playbook;
    • add env_file: .env to the service in the compose file.

  4. Add an nginx vhost in networking/nginx-config/<app>.abair.ie (see Public apps).

  5. Add the DNS record in Cloudflare, or ask an admin to add it if you don't have Cloudflare access. Mention it in the pull request so it isn't forgotten:

    • Type: A
    • Name: <app> (giving <app>.abair.ie)
    • IPv4 address: 136.243.40.234 (the Proxmox host, which forwards web traffic to the webserver)
    • Proxy status: Proxied (orange cloud)

    The site isn't reachable until this record exists, even after the deploy. See Add the DNS record.

  6. Commit to a branch, push, and open a pull request with an admin added as reviewer (see above). Wait for the admin to confirm it's deployed.

  7. Check your app's CD workflow, and re-run it if needed. Once the admin confirms the deploy, open the Actions tab of your app's own repo (on GitHub) and look at the latest run on main:

    • Passed: the latest image is deployed.
    • Failed: if it ran before the admin's deploy, its deploy step will have failed, because the app's directory (/opt/apps/<app>) didn't exist on the server yet. Re-run it (Re-run all jobs on GitHub, Re-run on Gitea). If it fails again after the admin's deploy, send the admin the error from the failed step.

    Then open the app in a browser at https://<app>.abair.ie to check it's working.

Changing an existing app​

  • Compose file: edit it on a branch, then open a pull request with an admin added as reviewer (see above).
  • Secrets: edit the <app> .env Secure Note in Vaultwarden. There's no pull request, so tell an admin it needs redeploying. If your code needs a new variable, follow Adding a new environment variable to an app instead. The order matters.
  • Code only: your app's own CI builds and deploys a new image (see CI/CD). Nothing in infrastructure-automation needs to change.

Adding a new environment variable to an app​

Use this when your code change needs a variable that the app's .env file doesn't have yet (a new API key, URL, feature flag, and so on).

The variable must be live on the server before your code reaches main

Pushing or merging to main in your app's repo deploys to production straight away. CI builds the new image, then runs docker compose pull && docker compose up -d on the server. The container starts with whatever .env file is already on the server.

That file only changes when an admin deploys it. Editing the note in Vaultwarden doesn't change the server, and neither does opening or merging a pull request in infrastructure-automation. If your code reaches main before the admin has deployed, production runs without the variable and the app breaks.

Before you start: how does your app read its .env?​

Open ansible/hetzner-main/services-user-apps/<app>/docker-compose.yml (or services-staff-apps/<app>/) and check the service:

What the compose file hasWhat you need to do
env_file: .envSteps 1, 3, 4 and 5. Every line in the note is passed to the container, so no infrastructure-automation change is needed.
An environment: list using ${VAR} (for example abair-api)Steps 1, 2, 3, 4 and 5. You also need to add the new variable to that list.
Neither, and there's no <app> .env note in VaultwardenThe app doesn't have a .env yet. Follow step 3 of Adding a new staff app or Adding a new user app first, then come back here.
Build-time variables are different

Variables baked into a frontend at build time (Next.js NEXT_PUBLIC_*, Vite VITE_*) are not read from the server's .env. They're set in your app repo's CI build, so this procedure doesn't apply. Add them to the CI config in the same pull request as the code that uses them.

Steps​

  1. Add the variable to the Vaultwarden note. In vault.abair.ie, open the Secure Note named <app> .env and add a new NAME=value line. Don't rename or remove existing lines. The code currently running in production still needs them. Save the note.

    Adding a variable is safe to deploy straight away: the code already in production just ignores a variable it doesn't use.

  2. Only if the compose file uses an environment: list: add the variable to it.

    environment:
      - NEW_VARIABLE=${NEW_VARIABLE}

    Commit this to a branch in infrastructure-automation, push it, and open a pull request with an admin added as reviewer (see above).

  3. Ask an admin to deploy, and include everything they need. For example:

    I've added NEW_VARIABLE to the <app> .env note in Vaultwarden. Pull request: <link, or "none needed">. The app is on services-user (or services-staff). Please merge the PR if there is one and deploy. My code that needs it is ready to merge once you confirm.

  4. Wait for the admin to confirm it's deployed. Don't skip this. A saved Vaultwarden note or an open pull request doesn't mean the server has the variable.

  5. Now merge or push your code to main in your app's repo. CI deploys it, and the container starts with the new variable.

Renaming or removing a variable​

Don't do this in one step: production would lose the old name before your new code is running. Instead:

  1. Add the new name alongside the old one, following the steps above.
  2. Once your code using the new name is in production, remove the old line from the note.
  3. Ask an admin to deploy again.

If production is already broken because of a missing variable​

  1. Add the variable to the <app> .env note in Vaultwarden (and open the compose pull request if step 2 applies).
  2. Ask an admin to deploy it as soon as possible. The deploy writes the new .env file and restarts the app with it. You don't need to push your code again.